Cross Validation

• Cross-validation (CV) is a way to choose optimal hyperparameters using the training data
• Split the training data into subsets, e.g., A, B, C, D, E
• Define a finite set of hyperparemeter combinations (a grid) to choose from
  • Example: {“max_depth”: [3, 4], “learning_rate”: [0.05, 0.1]}
  • Example: {“hidden_layer_sizes:[[4, 2], [8, 4, 2], [16, 8, 4]]}

• Use one of the subsets (e.g., A) as the validation set
• Train with each of the hyperparameter combinations on the union of the remaining subsets (e.g., B \(\cup\) C \(\cup\) D \(\cup\) E)
• Compute the trained model scores on A
• Repeat with B as the validation set, etc.
• For each hyperparameter combination, end up with as many validation scores as there are subsets

• Average the validation scores to get a single score for each hyperparameter combination
• Choose the hyperparameters with the highest average score
• All of this together is “search over the grid using cross-validation to find the best hyperparameters”
• It is implemented by scikit-learn’s GridSearchCV function

Custom scoring function

• If we want to use predictions to sort stocks into “good” and “bad”, then MSE is a poor scoring method.
• Example: if you buy a stock because the predicted return is 40% and the actual return turns out to be 100%, is that really an error?
• We want higher predictions to correspond to higher actual returns, so rank correlation would be a reasonable scoring method.

Coding a custom scoring function

from scipy.stats import spearmanr
from sklearn.metrics import make_scorer
score = make_scorer(spearmanr, greater_is_better=True)

Example

• Same data as in 3a-trees
  • agr, bm, idiovol, mom12m, roeq
  • training data = 2021-12
  • test data = 2022-01
• Quantile transform features and ret in each cross-section

Cross validate gradient boosting

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import GradientBoostingRegressor

model = GradientBoostingRegressor()

Cross validate AdaBoost

from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import AdaBoostRegressor

model = AdaBoostRegressor(
    base_estimator=DecisionTreeRegressor()
)